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Abstract 

For over 30 years tertiary institutions have used anonymous Likert type student feedback 
response forms to assess quality of teaching. Institutions have increasingly used these Student 
Evaluations of Teaching (SETs) to also inform tenure and promotion decisions. Accountability 
requirements of Quality Assurance have now made the practice almost universal. Yet, it is widely 
recognized that these combined uses of SETs result in significant problems including ‘grade inflation’ 
and limitations on academic freedom. It has also been demonstrated that lecturers can use simple 
techniques to increase their SET scores without increasing the quality of their teaching. It seems that 
SETs have continued to be used because there has so far been no cost effective and efficient 
alternative method of measuring teaching quality. 

This paper reports an inexpensive and efficient alternative method of measuring teaching quality. 
The method aligns students’ in-course expectations for change with their lecturer’s expectation for 
change in three process objects; namely Skills, Understanding and Attitudes. It is shown, using sample 
course data, that these in-course alignment indicators predict Quality Teaching, which is measured 
after the course using outcomes of students’ academic achievements and course enjoyment. 

Traditional SETs are post-mortem measures that do not feedback to help the students on the course. 
However, a major contribution of the alignment method is that optimum in-course changes can be 
calculated that minimise the in-course alignment scores, for the whole course or for any student sub- 
group of interest, and thus maximise the students’ predicted attainments. Hence, as the method only 
takes five minutes to administer, it can also be introduced at the level of the individual instructor who 
wishes to keep his or her teaching on track. 

The alignment method uses a single form and results in one decision point number. Thus, it can 
be used by the administration at the end of courses for comparable promotion and tenure decisions 
across the institution. Other benefits of the method will be discussed. The method is being continually 
up-graded and web-based alignment software is now being developed to enable lecturers and 
administrators from tertiary institutions worldwide to avail themselves of the method. 



Introduction 

This paper briefly reports an alternative system for assessing quality teaching in tertiary institutions and 
focuses on the use of student feedback. The traditional method of assessing quality of teaching has been by 
questionnaires that ask students to anonymously rate the quality of teaching on a 4 or 5 point Likert scale from 
strongly disagree to strongly agree. In the literature, the use of these forms is called Student Evaluations of 
Teaching (SETs). 

SETs have been used in universities for more than thirty years as part of the Quality Assurance Cycle to 
assess the quality of teaching and as an indicator of successful teaching for promotion, award and tenure 
decisions. Unfortunately, their use has been accompanied by many counter-productive effects such as 
discouraging innovative teaching, and deterring instructors from challenging students (Damron, 1995; Murray, 
1984; Ruskai, 1996). Although their outcomes are intended to improve teaching, a major negative effect of also 
using them to varying degrees for promotion and tenure decisions has been to contribute to the lowering of 
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academic standards. Results of analyses of SETs and expected grades suggest that instructors can “buy” better 
evaluations via more lenient grading (Krautmann, & Sander, 1999). In the copious literature on the subject, this 
effect is referred to as ‘grade inflation’ or ‘dumbing down courses’ and some universities who use SETs now 
make statistical adjustments for these effects (Gillmore, & Greenwald, 1999). SETs have become known as little 
more than ‘smile sheets’ measuring popularity and ‘customer satisfaction’ (Altschuler, 1999), and lecturers have 
developed many methods for improving their SET scores that do not necessary improve their teaching 
(Crumbley, 1995). Its seems that one reason SETs continue to be used is that there has not been an expedient 
alternative. This paper reports such an alternative - the Alignment method. 

The paper demonstrates an alternative method of using students’ evaluations of teaching (SETs) that 
circumvents many of the problems associated with traditional SETs. In particular, it shows how in-course 
feedback, consisting of eight ratings, can be used to optimize post-course academic attainment. The method is 
illustrated with data from course feedback at the University of the West Indies. 

Traditional Student Evaluations of Teaching (SETs) are feedback forms returned by students near the close 
of a course. Institutions intend that data from these forms be used to improve quality of teaching and as an 
assessment of quality of teaching for deciding faculty promotion, award and tenure decisions. Although it is 
recognized that students can offer valuable information on the appropriateness of teaching quality, it has also 
been recognized that these traditional SETs are likely to have negative effects on the quality of teaching. These 
negative criticisms are quite extensive and range from what is referred to as ‘dumbing down’ of courses to 
restrictions on academic freedom (Gillmore, & Greenwald, 1999; Simon, 1996). One patently obvious criticism 
is that the information given by one group of students at the end of a course cannot be used to improve the 
teaching on that course for the very students to whom the feedback applies. Similarly, it can only be useful to 
future students to the extent that future groups of students are similar to the feedback group and to the extent 
that the course and teaching remain similar. However, courses and teaching methods hopefully evolve and the 
constituent subgroups of a student cohort can change considerably from one teaching of the course to the next. 

This paper introduces an alternative method of allowing students to assess the quality of teaching that 
circumvents many of the problems associated with traditional SETs. In particular it allows feedback to be used 
for optimizing teaching quality during the course for the whole class, for individuals or for identified sub-groups 
of students within the whole group. The feedback is efficient to process - as it requires only eight ratings from 
each course member. This more timely feedback and optimization allows teaching choices for keeping teaching 
on track for the very students who are giving the feedback. An added advantage is that the method can be used 
by administration at the close of a course to calculate a single indicator of teaching quality that can be used for 
comparative promotion, award and tenure decisions across the institution. The paper explains how the use of 
this indicator protects lecturers from students’ intransigence and protects the institution from ‘dumbing down’ 
of courses. 

The paper outlines the method and the theory behind it. The reliability and validity of the method is 
demonstrated with actual data from a course assessment. The method operationally defines three educational 
process objectives - Skills, Understanding and Attitudes. These three objectives are emphasized to a determined 
amount in the teaching and assessment of courses using this method. Feedback forms used during the course 
give data on the lecturer’s and students’ expectations for change in the emphasis of these objectives. This data 
allows for calculations of the alignment between the lecturer’s and the students’ expectations for change. The 
Alignment theory presented here is that students’ academic success and enjoyment of teaching are maximized 
when students and their lecturer are working towards the same changes. The theory is re-validated with each 
course by correlations of alignments with results, which show that in-course alignment predicts post-course 
academic success. This paper describes how the data are also used during the course to determine the changes 
that will best align in-course student/lecturer expectations and so maximize predicted post-course academic 
attainment for the whole group or for different student sub-groups who are taking the course. 

The educational importance of this Alignment method is that it offers an efficient and effective alternative 
O ,ie widespread problematic use of traditional SETs for quality control of teaching in tertiary institutions. 
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The Alignment method 

Cohen (1994) has introduced to education the term ‘instructional alignment’, meaning the alignment of 
teaching, assessment and objectives. Instructional alignment has been found to compare favourably with the use 
of other commonly used strategies intended to improve learning performance, such as criterion-referenced tests, 
curriculum-based measurement, direct instruction, learning strategies, peer tutoring, self-instructional training, 
cooperative learning and computer-assisted instruction (Ippolito, 1990; Redding, 1992; Vergason, & Anderegg, 
1991). Elia (1994) and Walker (1998) found that instructional alignment had an unusually high positive learning 
effect producing substantial improvement on the achievement levels of disadvantaged and low performing low 
socio-economic level school students. Similarly, results of a controlled experiment with community college 
learners comparing verbal mediation, feedback monitoring, and instructional alignment, showed that alignment 
had the greatest overall effect on achievement (Breitsprecher, 1991). Biggs (1999) highly recommends 
educational alignment for improving the quality of university teaching. It has been found that aligning teaching, 
assessment and objectives in tertiary education can increase learning performance by up to two standard 
deviations (Cohen 1987, 1991; Cohen, Hyman, Ashcroft, & Loveless, 1989). 

There are many psychometric instruments that use what is referred to here as ‘Alignment methods’. In an 
Alignment method a respondent’s current state is assessed and his/her ideal state is also assessed on the same 
indicators. The difference between the current and ideal states is the measure of alignment. Where the difference 
is large, there is poor alignment which is indicative of problems. Where the difference is small, alignment is 
good, which indicates that the current state is close to the ideal. Improved alignment can also be used as a 
measure of successful intervention. What is crucial to the Alignment method is the choice of indicators 
measured to assess the current and ideal states. For each course, Cohen (1987, 1991) used three different 

objectives that were derived from the course instructor’s idiosyncratic values. However, to allow 
comparisons, the Alignment method introduced here, following Bloom’s (1 956) taxonomy of educational 
objectives, defines three standard process objectives that are emphasized to different degrees in quality teaching 
and learning. 

These are Skills, Understanding and Attitudes operationally defined here as: 

1 . Skills - learning of facts/processes. Assessed by speedy accurate reproduction. 

2. Understanding - professional competence. Assessed by justification of novel application and 

3. Attitudes - professional values. Assessed by demonstration. 

The method uses alignment on these process objectives as indicators of quality teaching. It should be noted 
that critical thinking is expected to be promoted by teaching and assessment of professional competence. This is 
because answers are not assessed as being right or wrong; only justifications that offer evidence of critical 
thinking are assessed. Alignment is not based on the course objectives but on the three process objectives. This 
is similar to common applications of Bloom’s taxonomy, which also emphasise different levels within domains. 
The Alignment method uses course objectives and content as vehicles for emphasizing Skills, Understanding 
and Attitudes to the degrees deemed appropriate for the course. This emphasis will vary and reflect the course 
level and culture of the subject taught. 

What are aligned are ‘changes expected by the lecturer’ and ‘changes expected by the students’ in each of 
these three process objectives. Numerically stated: Alignment = changes expected by Lecturer - changes 
expected by students. Zero is the perfect score, the theory being that students achieve higher standards if they 
and their lecturer are working towards the same changes. Figure 1 shows the eight core questions that the 
lecturer and each student answers for the alignment to be calculated. 

These forms are confidential, not anonymous. When students enrol, the process objectives are explained 
with generalised examples related to teaching and assessment. As part of their orientation they pass a test on 
their competency to evaluate in order to earn the right to be considered as informed assessors. At the start of 
each course, their lecturer gives subject specific examples as part of the introduction to the course. It has been 
found that exemplary university teachers find their own different effective teaching dimensions and strategies to 
cr-hieve excellence (Hativa, Barak, & Simhi, 1999). Hence, although staff development units may advise, the 
techniques for attaining these goals are left as a matter of informed professional choice to the lecturer. 

O 



ERJC hin § 



The Alignment method: Page 4:15 



Figure 1 : Five minute feedback form 



Course Assessment - Skills, Understanding and Attitudes 

Estimate, for you personally, how much this course emphasises, and should emphasise (i) Skills, 

(ii) Understanding and (iii) Attitudes? Do this for both how the course is now, and for how the course 
should be - write a number in each box. 



As it is now on this course 



(i) Emphasis on Skills 
(getting it right) 



Your 
estimate 
out of 100 



As it should be on this course 

, , Write a 

I Your Number 

estimate ^ each 

I lout of 1 00 }j OX 



As it is now on this course 



As it should be on this course 



(ii) Emphasis on Understanding 
(knowing why) 



Your 
estimate 
out of 100 



Your 
estimate 
out of 100 



Write a 
Number 
in each 
box 



As it is now on this course 



As it should be on this course 



(iii) Emphasis on Attitudes 
(relevance to your life) 



Your 
estimate 
out of 100 



Your 
estimate 
out of 100 



Write a 
Number 
in each 
box 



Flow much so far have 

you enjoyed your experience 

of the teaching on this course? 



Your 
estimate 
out of 100 



Write a 
Number 
in each 
box 



Before you started this course, how much did you expect that you 
would enjoy your experience of the teaching on this course? 



Your 
estimate 
out of 1 00 



Using data from the forms, individuals’ alignments can be calculated and grouped to calculate and 
compare the mean alignments of any student sub-group of interest - males v females, experienced v novice 
students, older v younger students, option 1 v option 2 students, etc. 

Two alignment scores are calculated; 

• Alignment of Scope (changes in absolute quantity of the three objectives) and 

• Alignment of Proportions (changes in relative quantity of the three objectives) 

These calculations are explained in detail later in this paper (Tables 2, 3„ 4 and 7) 

Flowever, as we shall see, these formative Alignment indicators, that are measured during the course, are 
only predictors of quality teaching. They are not the criteria of quality teaching. The Alignment method 
separates the measurement of these in-course predictors of quality teaching from the post-course measurement 
of its two criteria of quality teaching. The two post-course summative criteria of quality teaching are: 

• Students’ academic standards and 

• Students’ enjoyment of the teaching 

This Alignment method, unlike the traditional SET ‘smile sheets’, distinguishes between the assessment of 
enjoyment and the assessment of academic standards (Naftulin, & Ware, 1973). 
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Validation of the theory 

When the courses are over and the academic results are compared with the alignment scores, it is possible 
to validate the theory for each course, and for each sub-group of students taking each course by correlating the 
‘Alignment of Scope’ with ‘Academic standards’ and by correlating the ‘Alignment of Proportions’ with 
‘Enjoyment of teaching’. 

Further, when the course has finished it is possible to use sensitivity analyses on the data to calculate those 
lecturer’s changes that would have most aligned the teaching and thus, according to the Alignment theory, 
maximized the academic results and enjoyment of the students. Table 1 shows the computer analysis of actual 
Alignment data from forms as in Figure 1, and each part of Table 1 is explained in the following sections, 
starting with Figure 2 that highlights the main sections of the table. It is seen from actual Alignment data 
illustrated in Table 1, that the choice of these preferred changes would have increased the correlations between 
alignment and academic standards, thus further validating the theory that alignments are predictors of quality 
teaching. This is now described in detail. 

Illustrative Results 

Table 1 illustrates a typical computer data input, analysis and results sheet for the Alignment assessment of 
a course. These computer calculations are not normally shown as part of the method. Only diagnostic reports 
and summary results are normally reported. However, to explain the detailed workings of the method, these 
calculations are shown in this paper. The method is designed to be generalised across courses, lecturers, 
subjects, levels, institutions and cultures. However, this illustrative data is from a postgraduate teacher 
education course called ‘Psychology for teachers’ that was taken by 36 mainly mature students. 

Figure 2 highlights the three main sections of Table 1, which is in three parts, Part 1 ‘For the Lecturer’, 

Part 2 ‘For the students’ and Part 3, the ‘Summary Results’. Parts 1 and 2 hold data from the Lecturer’s form 
and the Students’ forms respectively. This data can be entered and analysed during the course to calculate 
Alignment predictors and optimal changes the lecturer should make to maximize students post-course academic 
attainment. Part 2 also has a section where the students’ post-course academic results are entered so that the 
Alignment predictions can be validated. Part 3 holds results of calculations from the Lecturer’s and Students’ in- 
course data that show current in-course alignments resulting from the current changes that the lecturer is 
working towards. It also holds the optimal changes that the lecturer should work towards to maximise student 
attainment. 

In Table 1 , the results printed in bold type are the results of sensitivity analyses that minimise Alignments. 
In Part 1 of Table 1, the ‘Start’ row shows the first seven numbers input from the Lecturer’s Alignment form 
displayed in Figure 1 . The eighth number was not used for this analysis. Part 2 ‘For the Students’ shows just a 
selection of four rows for students numbered #12, #25, #35 and #30, from the Alignment forms of all 36 
students in this course. As well as the first seven numbers from the students’ Alignment forms, these rows have 
been extended to show other variables for the identification of sub-groups of interest. The last column for the 
students shows their academic results. These were entered after the course and are used to validate the 
predictions from the in-course Alignment indicators and to further validate the Alignment theory. The 
‘Summary Results’ section, Part 3, shows the calculated changes resulting from the lecturer’s start position. 
There were six option groups in this class. Part 3 also shows results for lowest and highest attaining option 
groups - for Option 2 ‘History’ students and for Option 4 ‘Modern language’ students, respectively. All the 
computations are done by computer, but for elucidation the detailed calculations contributing to these results are 
illustrated in Tables 2, 3, 4 and 7. 

Calculating Alignment of Scope 

Table 2 shows how the six raw ratings of the process objectives given by the lecturer and the 
corresponding six raw ratings given by student #12 are used to calculate the Scope alignment for that student, 
jg-- lecturer’s raw ratings of 30, 40, 40, 45, 60 and 50 are taken from Table 1 Part 1 ‘For the lecturer’. The six 
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Figure 2: Schematic illustrating the main sections of Table 



Part 1 For the Lecturer 



Data from the Lecturer's form 



ALIGNMENT ASSESSMENT - DATA SHEET OPTIMISATION OF TEACHING 
Part 2 For the Students 



Data from the Students' forms 
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Part 3 Summary Results 



Lecturer's changes in the three 
process objectives 



In-course 

Alignment 

predictors 



Post-course 

validations 



Table 1 : Example of computer input, analysis and results for the Alignment method 



Parti For the Lecturer 


Course 


ED40C 


Date 


15/11/00 


Lecturer's Name ... 




Skills 


Understanding 


Attitudes 


Enjoyment 


Student Variables of Interest 


Academic Results for 
Post-course Validation 


is now 


should be 


is now 


should be 


is now 


should be 


Sex 


Age 


Option 


Years Teaching 


Start 


30 


60 


40 


45 


60 


50 


50 


Best 


30 


30 


40 


44 


60 


71 




ALIGNMENT ASSESSMENT - DATA SHEET OPTIMISATION OF TEACHING 


Part 2 For the Students 


# in class 


36 


# present 


20 






Skills 


Understanding 


Attitudes 


Enjoyment 


# 


is now 


should be 


is now 


should be 


is now 


should be 


12 


50 


50 


95 


98 


90 


95 


95 


2 


30 


2 


6 


56% 


25 


35 


70 


60 


80 


80 


88 


55 


2 


25 


2 


3 


57% 


35 


80 


40 


20 


60 


40 


50 


30 


2 


25 


2 


3 


43% 


30 


40 


80 


70 


90 


70 


90 


75 


2 


21 


3 


0 


71% 




Part 3 


Summary Results 


Lecturer's changes 


Alignment 


Mean post- 


Validation 




Scope 


Proportion 


Predictors 


course results 


Correlations 




Skills 


Underst 


Attitudes 


Skills 


Underst 


Attitudes 


MScope 


MProp 


Enjoy 


Acad 


Scope 


Prop 




Whole 


Class 






n=20 








Start 


100% 


13% 


-17% 


68% 


-6% 


-30% 


1.499 


1.259 


69.5% 


63.0% 


-0.265 


-0.278 


Best 


0% 


10% 


19% 


-9% 


1% 


8% 


0.674 


0.507 






-0.307 


-0.576 




Option 


2 


History 




CO 

II 

c 














Start 


100% 


13% 


-17% 


68% 


-6% 


-30% 


1.861 


1.658 


60.0% 


52.2% 






Best 


0% 


33% 


9% 


-11% 


18% 


-4% 


1.226 


1.226 












Option 


4 


Modern Languages 


n=4 














Start 


100% 


13% 


-17% 


68% 


-6% 


-30% 


1.200 


0.972 


72.5% 


75.7% 






s - 3 S t 


13% 


2% 


6% 


6% 


-5% 


-1% 


0.223 


0.203 
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raw ratings given by student #12 of 50, 50, 95, 98, 90 and 95 are taken from Table 1 Part 2 Tor the student’. For 
ease of comparison of how Scope and Proportion Alignments are calculated, the same raw ratings from student 
#13 are also used in Table 7 to explain the calculation of alignment of Proportion. 



Table 2: Calculation of Scope Alignment 



#1? LECTURER'S RATINGS STUDENT'S RATINGS ALIGNMENT 



Scope 


IS NOW 


SHOULD 

BE 


SKILLS 


30 


60 


UNDERSTANDING 


40 


45 


ATTITUDE 


60 


50 



CHANGE 


IS- NOW 


SHOULD 


(L) 


BE 


100% 


50 


50 


13% 


95 


98 


-17% 


90 


95 



CHANGE 

(S) 


S-L 


ABSOLUTE 


0% 


-100% 


100% 


3% 


-9% 


9% 


6% 


22% 


22% 




TOTAL 

Scope 


1 32% 



The changes in Scope expected by the lecturer, and those expected by each student, for each process 
objective are calculated by dividing the difference between the raw ratings of the ideal state ‘should be’ and 
current state ‘is now’ by the raw ratings for the current state. 

Using the raw ratings: Change in Scope = (‘should be’ - ‘is now’) / ‘is now’ 



It is seen from Table 2, for example, that the lecturer’s starting position for Skills of 30 for ‘is now’ and 60 
for ‘should be’ requires a 100% increase, i.e. (60 - 30)/30= 1 00%. The student’s starting position for Skills of 50 
for ‘is now’ and 50 for ‘should be’ requires a change of (50-50)/50=0%. It should be noted that the difference is 
‘grounded’ by dividing by the starting position, resulting in a percentage increase. Just as $100 can mean more 
to a poor person than to a rich person, this division is intended to compensate for the diminishing expectations 
represented by the same differences at higher parts of the scale. For example, an increase of 1 0 points from a 
low starting position of 5 up to 15 implies greater ‘expectation’ that an increase of the same 10 points from a 
higher starting position of 50 up to 60; namely (1 5-5)/5=200% compared to (60-50)/50=20%. 

The Alignment of each process objective is the difference in change expected by the Student (S) and the 
change expected by the Lecturer (L); Alignment =S-L. For example, Table 2 shows that the Skills alignment for 
student #12 is 0%- 100%=- 100%. The negative sign indicates that the student is below the lecturer’s expectation 
for Skills. Flowever, the student’s positive Attitude alignment of 22% indicates that the student is ahead of the 
lecturer’s expectation for Attitude. Two further demonstrations of the calculation of Scope alignment are given 
in Figures 3 and 4. When the Alignment form is used in-course, these positive and negative process alignments 
can be reported individually or as sub-group statistics for the basis of diagnostic reports to guide teaching. 
However, it is the absolute alignments on the three process objectives that are summed to give the total Scope 
alignment for this student. The absolute values are chosen to be summed to avoid the possibility of positive and 
negative alignments cancelling each other and producing a result that would hide the degree of misaligned 
expectations. Again, absolute alignments can be reported individually or as sub-group statistics for the basis of 
diagnostic reports to guide teaching. However, it is the totals of the absolute Scope alignments that are used as 
predictors of academic attainment for individuals, for student sub-groups and for the group as a whole. The 
mean absolute Scope alignment for the whole group is the main indicator of quality teaching and predicts the 
academic attainment for the group. 
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Validating Scope alignment as a predictor of Academic Attainment 

Alignment theory predicts that students will achieve higher academic attainments if they and their lecturer 
are working towards the same changes. Thus, to validate the theory we expect the more aligned students, the 
ones with the lower in-course Alignment scores, to have the higher post-course academic results. This can be 
tested for each course that uses the Alignment method simply by correlating in-course Alignment scores with 
post-course academic attainment. It can also be validated for student sub-groups by comparing the mean Scope 
alignments and mean academic attainments of the groups. 

Post-course academic attainments for the illustrative data in Table 1 have been entered for individual 
students in the last column of Part 2 ‘For the students’. When we calculate the Scope alignment of the students 
we expect to find that their alignment and attainment is inversely related. Tables 3 and 4 give further examples 
of the calculation of Scope alignments for students #30 and #35 who have the highest and lowest academic 
attainments, respectively shown in Table 1 . 



Table 3: Calculation of Scope Alignment for student #30 



#30 LECTURER'S RATINGS STUDENT'S RATINGS ALIGNMENT 



Scope 


IS NOW 


SHOULD 

BE 


CHANGE 

(L) 


IS NOW 


SHOULD 

BE 


CHANGE 

(S) 


S-L 


ABSOLUTE 


SKILLS 


30 


60 


100% 


40 


80 


100% 


0% 


0% 


UNDERSTANDING 


40 


45 


13% 


70 


90 


29% 


16% 


16% 


ATTITUDE 


60 


50 


-17% 


70 


90 


29% 


45% 


45% 



TOTAL Scope 
ALIGNMENT D 1 /o 



Table 4: Calculation of Scope Alignment for student #35 



#35 LECTURER'S RATINGS STUDENT'S RATINGS ALIGNMENT 



Scope 


IS NOW 


SHOULD 

BE 


CHANGE 

(L) 


IS NOW 


SHOULD 

BE 


CHANGE 

(S) 


S-L 


ABSOLUTE 


SKILLS 


30 


60 


100% 


80 


40 


-50% 


-1 50% 


150% 


UNDERSTANDING 


40 


45 


13% 


20 


60 


200% 


1 88% 


188% 


ATTITUDE 


60 


50 


-17% 


40 


50 


25% 


42% 


42% 



TOTAL Scope qq . 

ALIGNMENT J/a/0 



Now that we have demonstrated the calculation of in-course Scope alignments for the three students #12, 
#30 and #35 we can compare them with the post-course academic attainments of these students as entered in 
Table 4. These paired predictors and criteria for the three demonstration calculations are tabulated in Table 5 and 
their relationship, illustrated in Figure 3, is seen to be inverse as predicted by the theory. 

O 
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Table 5: In-course Scope alignments and post-course academic attainments for the three demonstrated 
calculations. 



Individual results 



Total Scope Academic 
Student Alignment Attainment 

#30 0.61 71% 

#12 1.32 56% 

#35 3.79 43% 



Figure 3: Relation between in-course Scope alignments and post-course academic attainments for 
demonstrated calculations. 




Individual Total Scope Alignments 



The means of post-course academic results for the whole class and for the two option groups have also 
been calculated and are also shown in Table 1 in Part 3 Summary Results. These are abstracted from Table 1 and 
are listed in Table 6. Their inter-relationship, illustrated in Figure 4, is inverse as predicted by the theory. 

Table 6: Mean Scope alignments and academic attainments for the whole class and sub-groups. 



Means 





Scope 


Academic 




Alignment 


Attainment 


Whole Class 
Sub-Groups 


1.499 


63.0% 


Option 2 


1.861 


52.2% 


Option 4 


1.200 


72.5% 



Figure 4: Relation between means of Scope alignment and means of academic attainment 
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The ‘validation correlations’ column in Part 3 of Table 1 shows that the overall correlation of attainment 
and alignment of Scope for the group is -0.265; that is, the closer the alignment then the higher the student’s 
academic result. This validates the prediction of academic attainment by alignment of Scope for the whole class. 
Further evidence validating Scope alignment as a predictor of academic attainment is also given in the section 
on sensitivity analysis below. 

Calculating Alignment of Proportion 

The second in-course formative indicator is the alignment of Proportion. This is a predictor of enjoyment 
of teaching, which is the second criterion of quality teaching. The alignment of Proportion is calculated in the 
same way as the alignment of Scope except that the two Proportions represented by each process objective are 
used in place of the raw ratings. This is illustrated in Table 7 for student #12. 

Table 7: Calculation of Proportion Alignment 



#12 LECTURER'S RATINGS STUDENT'S RATINGS ALIGNMENT 



Proportions 


IS NOW 


SHOULD BE 


CHANGE 

(L) 


IS NOW 


SHOULD BE 


CHANGE 

(S) 


S-L 


ABSOLUTE 




Rating 


Proportion 


Rating 


Proportion 




Rating 


Proportion 


Rating 


Proportion 








SKILLS 


30 


0.231 


60 


0.387 


68% 


50 


0.213 


50 


0.206 


-3% 


-71% 


71% 


UNDERSTANDING 


40 


0.308 


45 


0.290 


-6% 


95 


0.404 


98 


0.403 


0% 


5% 


5% 


ATTITUDE 


60 


0.462 


50 


0.323 


-30% 


90 


0.383 


95 


0.391 


2% 


32% 


32% 


Totals 


130 


1 


155 


1 




235 


1 


243 


1 




TOTAL Pro 
ALIGNMENT 


1 09% 



We can replace both the ‘is now’ and ‘should be’ raw ratings of any process objective by their Proportions 
of the total of these three ratings. For example, in Table 7 we see that the total of the lecturer’s three raw ratings 
for ‘is now’ is 30+40+60=130. Using this total the ‘is now’ Proportion for each process objective can be 
calculated. For Skills, the ‘is now’ Proportion is 30/130=0.231. For Understanding, the ‘is now’ Proportion is 
40/130=0.308 and for Attitude the ‘is now’ Proportion is 60/1 30=0.462. A calculation check is that the three 
Proportions should total to 1 e.g. allowing for rounding errors 0.231+0.308+0.462=1 , as shown. Similarly the 
lecturer’s raw ratings of how the three process objectives ‘should be’ can be replaced by their Proportions of the 
‘should be’ total of 60+45+50=150. So the lecturer’s raw rating of 60 for what Skills ‘should be’ is replaced by 
the Proportion for Skills of 60/150=0.387. The lecturer’s ‘should be’ Proportion for Understanding is 45/ 

1 55=0.290 and for Attitude the lecturer’s ‘should be’ Proportion is 50/1 55=0,323. The calculation check that the 
three Proportions sum to 1 gives us 0.387+0.290+0.323=1, as shown. 

The calculations for changes in Proportions and for alignments of Proportions are exactly the same as the 
calculations for changes in Scope and for alignments of Scope, except that they use the Proportions rather than 
the raw ratings. The lecturer’s expected changes in the Proportions of the process objectives are given by the 
Proportions for ‘should be’ minus the Proportions for ‘as is’, and the result divided by the Proportion for ‘as is’. 

Using the Proportions: Change in Proportion = (‘should be’ - ‘is now’)/‘is now’ 

It is seen from Table 7 that the change the lecturer expects in the Proportion of Skills is (0.387-0.23 1 )/ 
0.231=68%. The change that student #12 expects in the Proportion of Skills is (0.206-0.21 3)/0. 2 1 3=-3%. It is 
interesting to note that although the change in Scope of Skills expected by student #12 is 0%, because her raw 
ratings are equal for ‘is now’ and ‘should be’, this student’s expectation for change in Proportion of Skills is - 
3%, and not 0%, because the two equal raw ratings represent different Proportions of the total raw ratings for ‘is 
now’ and ‘should be’. 

O 
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The calculation of alignment of Proportion for each process objective is similar to the calculation for the 
alignment of Scope, namely ‘Change in Proportion expected by the Student (S)’ minus ‘Change in Proportion 
expected by the Lecturer (L)’: 

Alignment of Proportion = S-L 

For example, Table 7 shows that the alignment of Proportions of Skills for student #12 is -3%-68%=-71%. 
The minus sign indicates that the student’s expectation for change is below that of the lecturer. As with the 
alignments of Scope, descriptive statistics of these positive and negative alignments of Proportions are reported 
individually and by student-groups as the basis of diagnostic reports for the improvement of teaching. However, 
as with alignments of Scope, it is the absolute values that are summed to give the alignment of Proportion.. The 
absolute alignments of Proportion are in-course formative indicators that predict students’ enjoyment of 
teaching. 

Validating Proportion alignment as a predictor of enjoyment of teaching. 

Part 3 of Table 1 reports both the mean alignment of Proportions and the mean enjoyment of teaching for 
the whole class and for two of the option groups. These results are abstracted and listed in Table 8 and the 
relationship between them is illustrated in Figure 5, which again shows the inverse relation predicted by the 
theory. Although the sizes of the sub-groups were small, these comparative sub-group results are also in 
agreement with the Alignment theory. Using the Student Variables we can do similar analyses for any subgroup 
of interest. 



Table 8: Means of Proportion alignments and enjoyment of teaching for the whole class and sub-groups. 



Means 





Proportion 


Enjoyment 




Alignment 


of Teaching 


Whole Class 


1.259 


69.5% 


Sub-Groups 
Option 2 


1.658 


60.0% 


Option 4 


0.972 


72.5% 



Figure 5: Resulting relation between means of Proportion alignment and means of enjoyment of teaching 
as predicted by theory 




The ‘validation correlations’ column in Part 3 of Table 1 shows that the overall correlation of enjoyment 
and alignment of Proportions for the group is -0.278. This validates the prediction of enjoyment of teaching by 
alignment of Proportions for the whole class. Further evidence validating Proportion alignment as a predictor of 
— Myment of teaching is given in the section on sensitivity analysis below. 

ERIC 
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Sensitivity analysis (SA) for calculating changes that optimize teaching and learning 

The Alignment software uses a standard Excel linear programming algorithm to find those lecturer 
changes that result in the best/minimum alignment. There are many choices of what to minimise or maximise 
for this calculation and these depend on (i) the indicator of most interest, (ii) the malleability of student 
expectations and (iii) the lecturer’s freedom to adjust course expectations. Applications determining these 
choices are discussed in the ‘Discussion’ section of this paper. The first example in Table 1 minimises the mean 
Scope alignment of 1 .499 for all the students down to 0.674 and calculates the ‘Best’ changes that would give 
this minimised mean Scope alignment. The resulting ‘Best’ changes, in bold type, are shown in Part 3 of Table 1 
as 0 % for Skills, 10% for Understanding and 19% for Attitudes. Given the lecturer’s initial evaluation of the 
three process objectives for the whole class as 30, 40 and 60, these optimum changes imply that the lecturer 
should have been working towards 30+30x0%=30 (no change), 40+40x1 0%=44 and 60+60x1 9%=71 . These 
‘Best’ changes for the whole class are shown for joint comparison in Part 1 of Table 1 under the lecturer’s Start 
ratings. If the lecturer had worked to these ‘Best’ changes then the students’ Scope and Proportion alignment 
scores would have changed and the correlations of academic attainment with their new Scope alignments and 
Proportion alignments would have improved from -0.265 and -0.278 to -0.307 and -0.576 respectively. This 
sensitivity analysis again demonstrates (i) the predictive validity of the Alignment indicators, in that as Scope 
alignment and Proportion alignment improve their correlations with attainment and enjoyment also improve, 
and (ii) the validity of the theory that the better the Scope alignment then the higher is the mean academic 
attainment and the better the Proportion alignment then the more the students enjoy the teaching. 

A similar sensitivity analysis was done separately for the History and Modern Language option groups by 
minimising their mean Scope alignments to find the best changes for these two sub-groups. Table 1 Part 3 
shows that the best alignments for these two groups are 1 .22 1 and 0.223 respectively which would have been 
given by the optimum ‘Best’ changes in the three process objectives of 0%, 33% and 9% for the History 
students and 1 3%, 2% and 6% for the Language students. 

If the lecturer had made these ‘Best’ changes for minimising the Scope alignment, then the mean 
Proportion alignments for these two option groups would also have improved from 1.658 to 1.226 for the 
History students and from 0.972 to 0.203 for the Modern Language students, modelling the relationship between 
enjoyment and attainment, and predicting greater enjoyment and attainment for both of these student-subgroups. 

Traditional SETs are a ‘post mortem’ assessment, collected at the end of the course when it is too late to 
use this feedback to help the students who made the assessments. However, a lecturer does not have to wait 
until the course is over to obtain diagnostic feedback and to optimize teaching using the Alignment method. The 
data collected in-course can be processed by this same type of sensitivity analysis to calculate the optimum 
changes that should be made by the lecturer during the course to maximize the students’ post-course academic 
attainment and/or enjoyment of teaching. 

Administrative decision point assessment of quality teaching 

The lecturer may utilize the five-minute Alignment form many times during the course to keep his/her 
teaching on track. The administration uses it just once near the end of the course to calculate the final Alignment 
score for that lecturer’s quality of teaching. This results in a single decision point number that can be compared 
across the institution and used for promotion and tenure decisions. 

It will be noticed from Table 1 that the minimum alignment that is possible for this group is 0.674. 
Remember that the best alignment is the one closest to the perfect score of zero. The minimum possible 
alignment for the History students was much higher at 1.221 than for the Modern Language students at 0.223. 
These minimum possible alignment scores illustrate the best teaching/learning that is possible with these sub- 
groups of students and reflects the fact that students are not all equally amenable to required educational 
changes in Skills, Understanding and Attitudes. To give the lecturer some protection from such intransigence the 
decision point measure of quality teaching is taken as the actual alignment less this minimum/best possible 
alignment. For example, referring to the data given in Table 1, given the alignment score for the whole class is 
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1.499 and the best possible alignment for the whole class is 0.674, the Quality Teaching (QT) score for the 
whole class is 1.499-0.674=0.825. These results from Table 1 are summarised in Table 9. 

Table 9: Quality Teaching (QT) scores for whole class and for student sub-groups accounting for student 
intransigence. 



Whole Class 
Sub-Groups 
Option 2 
Option 4 



Mean Alignment Scores 



Actual Scope 
1.499 

1.861 

1.200 



Best Scope 
0.674 

1.226 

0.223 



QT Score 
0.825 

0.635 high QT 
0.977 low QT 



Mean Academic 
Attainment 
63.0% 

52.2% 

72.5% 



Table 9 illustrates a novel research application of the Alignment method. It measures, for the first time, the 
differential effort that teachers expend in teaching mixed ability students. Educators generally accept that 
teachers expend more effort teaching ‘less-able’ students than in teaching the ‘more-able’ students in their 
classes. It is thought that the ‘more-able’ students, being more independent, can manage more on their own. For 
example, as students improve they become less dependent on teacher assistance (Davis, 1998) and higher ability 
students use their time more productively while waiting for teacher assistance (de la Cruz, 1995). 

It can be noticed from Table 9 that the quality of teaching was skewed more towards the needs of the more 
intransigent group; Option 2 History students The quality of teaching experienced by the History Students 
(option 2, QT=0.635) was 0.342 better than that experienced by the Modern Language students (option 4, 
QT=0.977). Table 1 and Table 9 show that this greater teaching quality produced a mean academic attainment of 
only 52.2% compared to the mean academic attainment of 72.5% for the Modern Language students who 
experienced a 54% lower quality of teaching. 



Discussion 

This article has introduced the Alignment alternative to traditional student evaluations of teaching (SETs). 
The method aligns changes in emphases expected by the students and their lecturers in the three operationally 
defined process objectives of Skills, Understanding and Attitudes. The theory is that students will achieve 
higher academic standards and enjoy the course teaching more if they and their lecturers are working towards 
the same changes. This Alignment is a proxy measure predicting two teaching outcomes. These are the two 
criteria of quality teaching that are used by the method; academic attainment and enjoyment of teaching. By 
using this proxy measure, the Alignment method avoids restrictions imposed upon the teaching process by the 
requirement of traditional SETs to operationally define Quality Teaching via the rating questions on the 
traditional SET forms. 

The Alignment method separates the measurement of these two outcome criteria so as to avoid the ‘smile 
sheet’ criticisms levelled at traditional SETs. It also separates the post-course measurement of these criteria 
from the measurement of in-course formative Alignment predictors of quality teaching. The in-course 
alignments are introspective ‘non-threatening’ ratings that students are pre-trained to use, whereas traditional 
SETs are one of the few evaluations affecting salaries and employment retention that are performed by untrained 
anonymous evaluators. Students are expected to pass a simple institutional test that qualifies them as raters. The 
short briefing for this test helps as an introduction to the lecturers’ briefing on their specific uses of the process 
objectives in the teaching and assessment of their courses. It accredits students as qualified raters and is 
intended to raise the importance, and quality of responses above that which low response rates and ill 
considered responses indicate that some students typically associate with the use of traditional SETs. The 5 
minute feedback form is relatively quick to administer and process. Because the form is confidential, and not 
anonymous, the results can be used to identify the quality of teaching experienced by individuals and by student 
sub-groups for whom the quality of teaching is a particular concern. 

Although this paper has detailed and justified the simple calculations of alignment for the purposes of 
elucidation, these calculations are normally computed by the Alignment software, which also uses the 
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components of the calculations to offer detailed diagnostic measurement based reports to improve the teaching 
experienced by identifiable individuals and sub-groups. Hence, the method can be used under the control of 
lecturers, usually two or three times in-course, to adjust their teaching and iron-out problems identified before 
the end of course assessment by the administration. The method results in a single decision point indicator of 
quality that can be used by the administration for equitable promotion, award and tenure decisions across the 
institution. The control, and targeted opportunities for improvement, that this method places in the hands of 
lecturers is intended to ameliorate feelings of threat and manipulation that can be engendered by traditional 
SETs. 

Unlike traditional SETs that are rarely validated, the Alignment method is validated on each course and 
each student sub-group with which it is used. It is validated by comparing, for example by correlating, the in- 
course predictive alignments with the post-course criteria of teaching, and this has been demonstrated in this 
paper. In addition, the theory is validated by sensitivity analyses that use standard linear programming software 
to show that the predictive correlations between the in-course indicators and post-course criteria of quality 
teaching improve when the alignments are minimised. The more aligned students and their lecturer were on 
Scope then the higher were the students’ academic results. Also, the more aligned students and their lecturer 
were on Proportion then the more the students enjoyed the teaching. These results agree with the theory. This 
same sensitivity analyses calculates the ‘Best’ possible changes in teaching that will optimally align teaching 
and learning to produce the highest academic attainments and enjoyment of teaching. A focus of this paper has 
been that this optimization of teaching can be done in-course. The sensitivity analysis can be used in-course to 
notify the lecturer of the changes in emphases of the three process objectives that would optimize the teaching 
experience for any student or student sub-group in the class. Another problem of traditional SETs is that they 
can not identify and compensate for low-ability over-confident students who blame the lecturer for their poor 
results. This paper has shown how the calculation of student intransigence protects lecturers from such 
‘blaming’ feedback. 

The precise information provided by the Alignment method identifies how to optimize alignment. 

However, if the changes corresponded to a reduction in expected quality of the course and these are 
implemented by the lecturer, then this will result in ‘dumbing down’ associated with traditional SETs. The 
Alignment method allows institutions to be protected from lecturers ‘dumbing down’ courses to improve their 
Alignment scores. This can be done by requiring pre-negotiated final ‘should be’ ratings that correspond to the 
institutions values of quality for that course, towards which the lecture should work. Similarly, the lecturer is 
protected from high institutional expectations that are impractical due to the intransigence of the students 
registered for the course. By making these expectations explicit and quantifiable the Alignment method protects 
both the institution and the lecturer. The paper has shown how intransigence, or teaching difficulty, is identified 
for each student, student sub-group and each course, and how this is allowed for in the calculation of quality 
teaching. This is particularly necessary where faculty are expected to teach subjects that challenge the values of 
their students. 

As the sensitive analysis gives the optimum changes required for the student cohort, these changes can be 
used to work backwards from the institutional expectations to calculate the minimum current threshold level at 
which the students need to be in order to reach the final institutional expectation for the course. This threshold 
level can be compared with the actual current student level, as computed from their eight ratings, and the 
difference used to assess the need to change the institutional expectations of course quality and/or the quality of 
the student intake to perhaps make quality teaching more viable for the course. 

It can be seen that the Alignment method uses the influence of assessment to improve both the quality of 
teaching and learning. In particular, it rewards lecturers for emphasising appropriate degrees of Understanding 
and Attitude, as well as Skills, in their teaching and assessments. It also rewards students for emphasising 
appropriate degrees of Understanding and Attitude, as well as Skills, in their learning and assessed assignments. 
For example, assessing Understanding by the justifications of novel application, as modelled in the teaching, is 
intended to promote critical thinking. 

O 
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